The invention relates generally to Information Technology (IT) devices, and in particular to an Integration Manager and Natural Interaction Processor (IMNIP) that addresses the need for embedded Information Technology (IT) devices to recognize user commands that are expressed using natural language and gesture. The natural language commands may be input directly by the user or generated by intelligent switches or sensors.
The inventor of the present invention has spent the last twenty-five years developing user interfaces that make it easier for disabled and aging people to use information technologies (IT) such as computers, telephones, and electronic control devices. While many of the devices encountered by these people perform similar tasks, they present the user with different interfaces and interaction strategies that make it necessary for the user to learn how to work with each one. This can be overwhelming for many people, particularly if they have memory or cognitive limitations. One of the strategies for overcoming this type of problem is to create interfaces that respond to naturally spoken commands and gestures. Natural Language Processing (NLP) is an essential component for such interfaces. While there is a large body of work in the NLP field, existing implementations require large fast computers to achieve acceptable real-time performance. Initial intelligent user interfaces used a software agent based NLP system to translate free speech commands into specific commands for interacting with computers and IT. Practical problems were encountered with this solution due to the need for a fast processor with large amounts of RAM, slow processing speed, and high licensing costs. An analysis of these problems and the required performance led to the present invention of a new embeddable system that meets the required performance specification while eliminating problems with size, speed and cost. It also incorporates an Integration Manager (IM) to merge multiple streams of input data. Whereas the IM function required a separate software module in the original NLP system, the new inventive system automatically performs the integration of multiple inputs. Because it is able to integrate and process inputs derived from multiple input devices such as keyboards, speech recognizers and a variety of pointing devices, the new system is described as an Integration Manager and Natural Interaction Processor (IMP). In a conventional NLP system, each user command is received as a single complete string of text that is broadcast to a network of software agents. Each agent is programmed to recognize a keyword and synonyms for that word. An individual agent asserts a claim if it recognizes a word in the command string. After a short delay, all of the asserted claims are collected and analyzed to determine if the pattern of asserted claims matches any predefined patterns of agents that represent particular outputs. This is an inherently slow process due to the amount of sequential processing required to execute all of the individual agent programs and the large amount of string comparisons and message passing that is necessary. Additionally, the agent-based software requires a large amount of message buffering in RAM. This is a serious problem for embedded systems that use small low-cost microprocessors that have minimal amounts of RAM.
The newly invented IMNIP uses an entirely different strategy to achieve the required speed and compact code. It is no longer necessary to wait until a complete command is received before beginning to process it. The IMNIP decodes each character as it is received and produces an output as soon as an unambiguous result is detected. Instead of storing and processing large numbers of text strings in RAM, as was necessary in the agent-based system, the IMNIP uses pointers, tables and arrays that can be pre-compiled and stored in ROM.